Slider: Incremental Sliding-Window Computations for Large-Scale Data Analysis

Sliding-window computations are widely used for data analysis in networked systems. Such computations can consume significant computational resources, particularly in live systems, where new data arrives continuously. This is because they typically require a complete re-computation over the full window of data every time the window slides. Therefore, sliding-window computations face important scalability problems. In this paper, we propose techniques for improving the scalability by performing sliding-window computations incrementally. In this paradigm, when some new data is added at the end of the window or old data dropped from its beginning, the output is updated efficiently by reusing previously run sub-computations, avoiding a complete recomputation. To realize this approach, we propose Slider, a novel framework that supports incremental sliding-window computations transparently and efficiently by leveraging selfadjusting computations principles of dynamic dependence graphs and change propagation. We implemented Slider based on the Hadoop MapReduce framework with a declarative SQL-like query interface, and evaluated it with a variety of applications and real world case studies from networked system. Our results show significant performance improvements for large-scale sliding-window computations without any modifications to the existing application code.